Ranking Human and Machine Summarization Systems
نویسندگان
چکیده
The Text Analysis Conference (TAC) ranks summarization systems by their average score over a collection of document sets. We investigate the statistical appropriateness of this score and propose an alternative that better distinguishes between human and machine evaluation systems.
منابع مشابه
Using Gene Expression Programming to Construct Sentence Ranking Functions for Text Summarization
In this paper, we consider the automatic text summarization as a challenging task of machine learning. We proposed a novel summarization system architecture which employs Gene Expression Programming technique as its learning mechanism. The preliminary experimental results have shown that our prototype system outperforms the baseline systems.
متن کاملFeature expansion for query-focused supervised sentence ranking
We present a supervised sentence ranking approach for use in extractive summarization. Using a general machine learning technique provides great flexibility for incorporating varied new features, which we demonstrate. The system proves quite effective at query-focused multi-document summarization, both for single summaries and for series of update summaries.
متن کاملQuery-focused summarization by supervised sentence ranking and skewed word distributions
We present a supervised sentence ranking approach for use in extractive summarization. The supervised approach achieves domain independence by making use of a range of word distribution statistics as features, of the sort typically used for unsupervised domain-independent ranking. We present empirical trials on the DUC 2006 query-directed multi-document summarization task, and demonstrate that ...
متن کاملAssessing the Effect of Inconsistent Assessors on Summarization Evaluation
We investigate the consistency of human assessors involved in summarization evaluation to understand its effect on system ranking and automatic evaluation techniques. Using Text Analysis Conference data, we measure annotator consistency based on human scoring of summaries for Responsiveness, Readability, and Pyramid scoring. We identify inconsistencies in the data and measure to what extent the...
متن کاملGraph-Based Multi-Modality Learning for Topic-Focused Multi-Document Summarization
Graph-based manifold-ranking methods have been successfully applied to topic-focused multi-document summarization. This paper further proposes to use the multi-modality manifold-ranking algorithm for extracting topic-focused summary from multiple documents by considering the within-document sentence relationships and the cross-document sentence relationships as two separate modalities (graphs)....
متن کامل